Using crowd-sourcing for query classification and analysis

In order to gain a better understanding of users and their intent behind web searching activities, first steps involve the analysis of the query submitted by the user and correct categorical classification of the query as a input for further analyses. Natural Language Processing (NLP) is an area with many inaccuracies for problem areas in Word Sense Disambiguation (WSD) and terms detected that are beyond the scope of the verification source[16]. Other difficulties include named entity recognition where capitalisation and language anomalies cause issue. These classic information retrieval issues are all relevant to query classification and this proposal will outline another approach to classification that uses the intelligence and judgement of people to counter the aforementioned challenges. There is clearly potential to resolve tasks with crowd-sourcing that require complex judgement and have many input factors for consideration, but with web search queries, the subjectivity is less prominent depending on the query type. The work outlined here describes experimentation with the use of Amazon Mechanical Turk (MTurk) as a classification tool. 1200 queries have been carefully selected and an experiment will be set up in MTurk to classify those queries into categories derived from various sources. Once a given query has been classified (according to a minimum inter-ranked consistency for that query), the query will subsequently be subjected to classification into sub-categories of its confirmed, higher level category, eventually resulting in its removal from the system when its classification is definitive. This is a subset of work contributing to the study in personalisation of web search activities and is a preliminary stage of understanding user intent that is required for further advancement into evaluating current personalisation techniques and developing new or modified means of best delivering search results based on various personalised factors.